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Relevance to the HRP Risks 

and Gaps 

The goal of this research project is to contribute to the 

closure of Human Research Program (HRP) gaps relevant 
to the “Risk of Inadequate Design of Human and 
Automation/Robotic integration (HARI)”, by providing 
information on how display and control characteristics affect 
operator performance: 

- Gap SHFE-HARI-01: What guidelines and tools can we develop 
to enable system designers and mission planners to conduct 
systematic task/needs analyses at the appropriate level of detail 
to allocate work among appropriate agents (human and 
automation)? 

- Gap SHFE-HARI-02: How can performance, efficiency, and 
safety guidelines be developed for effective information sharing 
between humans and automation, such that appropriate trust 
and situation awareness is maintained? 


Human-Robot Interaction 


Multi-year research project investigating three areas 
applicable to NASA robot systems: 


CEV(L1 


1 ) The effects of video overlays on teleoperation of a robot arm and 
a mobile robot 


2) The effect of camera locations on a mobile vehicle for 
teleoperation 

3) The types of gestures and verbal commands applicable to 
human-robot interaction 


Slide 3 


CEV(L1 I would not necessarily call these three areas. Areas would be broader such as co- located and remote operation in which these three 
fit under 
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Video overlays 


Experimental and field research have revealed numerous issues 
related to the use of video as the primary means of controlling a 
robot. 

- Operator may need to perform mental translations and rotations to infer 
the control movements needed to teleoperate a robotic arm to a desired 
position (Smith & Stuart, 1989). 

- The use of video is constrained by environmental conditions (e.g., dust, 
darkness, and severe lighting conditions) and communication 
limitations, such as time delays, low bandwidth, and jitter. 

- Video may lack the perceptual cues that are used by humans when 
performing tasks such as driving or controlling a collocated robot (Tittle, 
et al., 2002; Woods, et al., 2004). 

Overlays can compensate for these issues 


Study 1 

Situation and command guidance overlays 
for teleoperation using a hand controller 

The study evaluated the effects of overlays on operator 

task performance during teleoperation of a robot arm. 

Three overlays were designed to compensate for the 

lack of perceptual cues by providing guidance: 

- Command guidance (CG) overlay - explicit instructions on what 
commands to input 

- Situation guidance (SG) overlay - implicit cues so that operators 
can infer the input commands. 

- The combination of CG and SG (SCG) provides operators with 
both explicit and implicit cues allowing the operator to choose 
which symbology to utilize. 


Study 1 
Method 

Design 

- Within-subjects design with practice and experimental trials 

- Order of overlay conditions counterbalanced 

Participants 

- Eighteen participants with little or no experience with teleoperation 

Task 

- Use the joystick to align the robot arm with the target and then grasp the target 
using the arm gripper without colliding with the target or the stand. 

Measures 

- Objective: 

• Task success rate 

• Overshoots 

• Task completion time 

- Subjective ratings after each condition: 

• Adequacy of information 

• Ease of accomplishing the task 

• Workload 

- At the end of the study, participants ranked the conditions. 



Study 1 

Overlay types/Conditions 




No guidance 



Command guidance 



Situation guidance 



Combined guidance 







Study 1 

Robot and joystick 




CoroWare CoroBot 



Top left button (O) to 
open the gripper 


Move arm up and down 
with the castle switch 


Top right button (C) to close 
the gripper 


Move arm forward and 
backward with the joystick 


Turn robot left and right with 
twisting the joystick 


Saitek Cyborg joystick 





Study 1 
Results 


Task success rate 

- No guidance and command guidance led to significantly better 
performance than the combined guidance overlay 

• Based on subject comments the combined condition had too much 
information and was confusing 
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The overlays had no significant effect on overshoots and 
task completion time. 





Results 

Subjective ratings 




Adequacy of information (5-point scale, 1 being not at all adequate 

• All overlay conditions were rated as more adequate than the no 
overlay condition 

- The combined overlay was rated significantly higher than the situation 
guidance overlay but not significantly higher than the command guidance 
overlay. 
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Results 

Subjective ratings 



Preference ranking: 5-point scale, 1 being most preferred, 4 being least 
preferred. 

- Command guidance was the most preferred, no overlay was the least 
preferred. 
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Overlay Type 
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The overlays had no significant effect on ratings of ease of 
accomplishing the task and workload. 


Situation and command 
guidance overlays - uplinked commands 


Limitations of Study 1 

- The task was too easy - participants 
performed well without any overlays 

- The situation guidance overlay did not 
serve as real situation guidance in the 
context of the task 

- Frequent malfunctioning of the robot 
resulted in data loss 

A follow-up study will improve the task 
and the overlays 

- New robot arm purchased 

- The study will use a hand controller and 
uplinked command as a within-subjects 
design 



Situation guidance overlay 



Cyton Gamma 300 with 7 DOF 
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This does not 
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Study 2 

Superimposed and integrated overlays 

When an overlay is superimposed on the external world, 
it appears to be fixed onto the display and internal to the 
operators’ workstation. 

Integrated overlays often appear as three-dimensional 
objects and move as if part of the external world. 

- Studies conducted in the aviation domain show that integrated 
overlays can improve performance compared to superimposed 
by reducing the amount of deviation from the optimal path 

- However, integrated overlays can cause “tunnel vision”, a 
decrease in situation awareness 

The purpose of the study was to investigate whether 
these results apply to HRI tasks, such as navigation with 
a mobile robot. 
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Study 2 
Examples 


Superimposed overlays 




Integrated overlays 






Study 2 
Method 



• Design 

- Within-subjects design with practice and experimental trials 

- Order of overlay conditions counterbalanced 

• Participants 

- Five participants with little or no experience with teleoperation 

• Task 

- Navigate the robot through the obstacle course as quickly and as 
accurately as possible without colliding with the obstacles 


Study 2 
Measures 



• Objective 

- Number of collisions 

- Average speed 

- Task completion time 

- Deviations from optimal path 

- Situation awareness measure: 

• Participant had to call out “Alien” when they saw a picture of it 
(among other pictures as distractors) 

• Subjective 

- After each condition, participants rated: 

• Amount of information provided 

• Ease of accomplishing the task 

• Workload 

- At the end of the study, participants ranked the conditions. 


Start 


Study 2 
Driving course 




Study 2 

Robot and joystick 




Drive robot forward 
backward 


Turn robot left and right by 
twisting 


TurtleBot 


Saitek Cyborg joystick 




Study 2 

Overlay conditions 



Superimposed overlay 



Integrated overlay 



Head-down overlay 




Study 2 

Preliminary results 

• The integrated overlay was ranked as the most preferred 
by most participants. 

• Limitations of Study 2 

- The task was too predicable 

- Situation awareness measure not sensitive enough 

- Participants could use the fiducial markers forjudging distance 

• Follow-on study 

- Revise the overlays to make them more similar to the “tunnel-in- 
the-sky” integrated overlays 

- Make the driving course more challenging 

- Use a better situation awareness measure 




